NLP Preprocesing

Here, we will performing pre-processing of documents using nltk, which is important before feature extraction phase

Downloading important resources

Before proceeding further, we need to install nltk library suite in our system using the command

pip install nltk

A pop-up window will open and we need to select All to download all the packages, corporas provided by nltk

Get a raw document from the nltk corpora

We will be retrieving a sample document from the nltk corpora, here we will be using the state_union corpus

Stop-word Removal

Here we are concerned about filtering the document from unnecessary words which include determiners, articles, ... But first we need to tokenize the document into words, and then filter by stop_word removal

Stemming

After stop-word removal, we perform stemming, which is basically a process of reduction of words into their root_word, without any consideration for meaning

As we can see from above that the word president is reduced to its root word presid

Lemmatization

Unlike stemming, lemmatization reduces a word to its root_word, such that the root_word has some meaning.

Here unlike stemming, the meaning of the lemmas are intact

Part of Speech Tagging

To maintain the context throughout, it is necessary to tag words by their parts of speech, so that we can filter them by tags to do analysis

The tag list and their meaning is described herewith

    CC coordinating conjunction
    CD cardinal digit
    DT determiner
    EX existential there (like: “there is” … think of it like “there exists”)
    FW foreign word
    IN preposition/subordinating conjunction
    JJ adjective ‘big’
    JJR adjective, comparative ‘bigger’
    JJS adjective, superlative ‘biggest’
    LS list marker 1)
    MD modal could, will
    NN noun, singular ‘desk’
    NNS noun plural ‘desks’
    NNP proper noun, singular ‘Harrison’
    NNPS proper noun, plural ‘Americans’
    PDT predeterminer ‘all the kids’
    POS possessive ending parent’s
    PRP personal pronoun I, he, she
    PRP$ possessive pronoun my, his, hers
    RB adverb very, silently,
    RBR adverb, comparative better
    RBS adverb, superlative best
    RP particle give up
    TO, to go ‘to’ the store.
    UH interjection, errrrrrrrm
    VB verb, base form take
    VBD verb, past tense took
    VBG verb, gerund/present participle taking
    VBN verb, past participle taken
    VBP verb, sing. present, non-3d take
    VBZ verb, 3rd person sing. present takes
    WDT wh-determiner which
    WP wh-pronoun who, what
    WP$ possessive wh-pronoun whose
    WRB wh-abverb where, when

Named-Entity Recognition

Apart from tagging words by their parts of speech, we can also group the words based on their context and meaning like grouping by name of persons, date , location, organistaion

To get more clarity we can use namedEnt.draw() to get the tree-structure of the named entities

Word embeddings

One hot vector

According to wikipedia,

Step 1:

• We import the numpy library for array manipulation.

Step 2:

• We take the set of unique words in our dataset

Step 3:

• We create a dictionary to map each unique word to an index

Step 4:

• We initialize a 2D matrix of all 0s for our entire set of one-hot vectors

Step 5:

• We set the exact position to 1 for each word as per our mapping method

Bag-of-Words (BoW)

According to wikipedia,

Step 1:

• We will first preprocess the data, in order to:
  • Convert text to lower case.
  • Remove all non-word characters.
  • Remove all punctuations.

Step 2: Obtaining most frequent words in our text.

We will apply the following steps to generate our model:

• We declare a dictionary to hold our bag of words.
• Next we tokenize each sentence to words.
• Now for each word in sentence, we check if the word exists in our dictionary.
• If it does, then we increment its count by 1. If it doesn’t, we add it to our dictionary and set its count as 1.

Sample count:

The entire BoW for our dataset:

TF-IDF

According to wikipedia,

First of all, we create the set of documents,

We import sklearn's TfidfVectorizer for fast idf calculation.

After that, we initialize an object and apply the fit_transform() method on our documents.

We define two auxiliary functions:

Finally, we visualize the tfidf for the first document.

Word2vec

Word2vec is a neural network based language model that converts words into vectors (embeddings) in a high-dimensional space.
This model was first introduced in this arxiv paper.

However, we would refer ch. 6 of Speech and Language Processing (3e draft) (by Dan Jurafsky and James H. Martin) and follow skip-gram with negative sampling (SGNS) method.
Link here.

At first, we import required modules.

For simplicity, we consider an embedding of 100 dimensions for our output.
We keep the window relatively small -> 2. That is, we would consider 2 words to the left, and 2 words to the right of the center word.
Plus, we randomly take 4 negative samples for every neighbor of a center word.

We initialize two types of weights for a single word $w$ -

• when $w$ is a center word
• when $w$ is in the context but not a center word

The procedure for our neural network forward function -

We compute the loss in each step using this formula -

Finally, we backpropagate the loss using this set of formulas -

Find embedding for a sample word -

Now a bit of visualization:

GloVe

GloVe (Global Vectors for Word Representation) is a word embedding model that represents words as dense vectors based on their co-occurrence patterns in a text corpus. It captures semantic and syntactic relationships between words, allowing words with similar meanings or contexts to be mapped to nearby points in a vector space.

This time, we would refer the GloVe paper.

Observing the proposed model, for each word $w$ in our vocabulary, we initialize 2 types of word embedding matrices, and 2 types of biases -

• when $w$ is a center word
• when $w$ is in the context but is not the center word

We maintain the same (100) embedding dimension.

Finally, we take the laid-out suggestion for the weight function -

BERT

BERT or Bidrectional Encoder Representations from Tranformers is designed to pre-train bi-directional representations from unlabelled text by jointly conditioning on both left and right context. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create a state of the art model for a wide range of NLP tasks. It was introduced and referenced in this paper.

BERT Architecture with 12 Encoder Blocks

Unlike word2vec or GLOVe which are context-free models that generate a single word embedding representation for each word in the vocabulary. An instance can be the word 'deposit' would have the same representation in 'sand deposit' and in 'bank deposit'.

Contextual models like BERT instead generate a representation of each word that is based on the other words in the sentence, by capturing word relationships bidirectionally.

There are four types of pre-trained versions of BERT depending on the scale of the model architecture:

1. BERT-Base (Cased / Un-Cased): 12-layer, 768-hidden-nodes, 12-attention-heads, 110M parameters

2. BERT-Large (Cased / Un-Cased): 24-layer, 1024-hidden-nodes, 16-attention-heads, 340M parameters

BERT relies on a Transformer (the attention mechanism that learns contextual relationships between words in a text). A basic Transformer consists of an encoder to read the text input and a decoder to produce a prediction for the task. Since BERT’s goal is to generate a language representation model, it only needs the encoder part. The input to the encoder for BERT is a sequence of tokens, which are first converted into vectors and then processed in the neural network.

Input embedding is a combination of 3 embeddings:-

1. Token embeddings: A [CLS] token is added to the input word tokens at the beginning of the first sentence and a [SEP] token is inserted at the end of each sentence.

2. Segment embeddings:A marker indicating Sentence A or Sentence B is added to each token. This allows the encoder to distinguish between sentences.

3. Positional embeddings:A positional embedding is added to each token to indicate its position in the sentence.

Pre-Training Tasks

1. Masked Language Modelling
2. Next Sentence Prediction

MLM(masked language modelling) is the task of predicting the next word given a sequence of words. In MLM, instead of predicting every next token, a precentage of input tokens are masked at random and only those masked tokens are predicted. Denoted by [MASK]

ABout 80% of the tokens are actually replaced with the token [MASK], 10% are replaced with random token, and rest 10% are left unchanged

Next Sentence Prediction is a binary classification task in which, given a pair of sentences, it is predicted if the second sentence is the actual next sentence of the first sentence. Easy, in case of a monolingual corpus as ours, and is helpful in minimising the loss function.

Implementation

Here however we will be using a pre-trained BERT model and tokenizer from the transformers library, as training the model from scratch requires high computational resources and time